Method and Device for Presenting Synthesized Reality Content in Association with Recognized Objects

ABSTRACT

In one implementation, a method includes: obtaining image data from an image sensor; recognizing a portion of an object within the image data; obtaining synthesized reality (SR) content associated with the portion of the object; and displaying the SR content in association with the portion of the object. In some implementations, the SR content is dependent on the orientation of an electronic device or the user relative to the object. In some implementations, the SR content is generated based on sensor data associated with the object.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.16/959,253, filed on Jun. 30, 2020, which claims priority toInternational Patent App. No. PCT/US2019/014288, filed Jan. 18, 2019,which claims priority to U.S. Provisional Patent App. No. 62/734,065,filed on Sep. 20, 2018, and U.S. Provisional Patent App. No. 62/620,339,filed on Jan. 22, 2018, which are hereby incorporated by referenceherein in their entireties.

TECHNICAL FIELD

The present disclosure generally relates to synthesized reality (SR)content consumption, and in particular, to systems, methods, and devicesfor presenting SR content in associated with recognized objects.

BACKGROUND

Virtual reality (VR) and augmented reality (AR) are becoming morepopular due to their remarkable ability to alter a user's perception ofthe world. For example, VR and AR are used for learning purposes, gamingpurposes, content creation purposes, social media and interactionpurposes, or the like. These technologies differ in the user'sperception of his/her presence. VR transposes the user into a virtualspace so their VR perception is different from his/her real-worldperception. In contrast, AR takes the user's real-world perception andadds something to it.

These technologies are becoming more commonplace due to, for example,miniaturization of hardware components, improvements to hardwareperformance, and improvements to software efficiency. As one example, auser may experience AR content superimposed on a live video feed of theuser's setting on a handheld display (e.g., an AR-enabled mobile phoneor tablet with video pass-through). As another example, a user mayexperience AR content by wearing a head-mounted device (HMD) orhead-mounted enclosure that still allows the user to see his/hersurroundings (e.g., glasses with optical see-through). As yet anotherexample, a user may experience VR content by using an HMD that enclosesthe user's field-of-view and is tethered to a computer.

BRIEF DESCRIPTION OF THE DRAWINGS

So that the present disclosure can be understood by those of ordinaryskill in the art, a more detailed description may be had by reference toaspects of some illustrative implementations, some of which are shown inthe accompanying drawings.

FIG. 1A is a block diagram of an example operating architecture inaccordance with some implementations.

FIG. 1B is a block diagram of another example operating architecture inaccordance with some implementations.

FIG. 2 is a block diagram of an example controller in accordance withsome implementations.

FIG. 3 is a block diagram of an example head-mounted device (HMD) inaccordance with some implementations.

FIG. 4 is a block diagram of an example optional display device inaccordance with some implementations.

FIGS. 5A-5B illustrate example synthesized reality (SR) presentationscenarios in accordance with some implementations.

FIGS. 6A-6B illustrate example SR presentation scenarios in accordancewith some implementations.

FIG. 7 illustrates an example SR presentation scenario in accordancewith some implementations.

FIG. 8 is a flowchart representation of a method of presenting SRcontent in associated with recognized objects in accordance with someimplementations.

In accordance with common practice the various features illustrated inthe drawings may not be drawn to scale. Accordingly, the dimensions ofthe various features may be arbitrarily expanded or reduced for clarity.In addition, some of the drawings may not depict all of the componentsof a given system, method or device. Finally, like reference numeralsmay be used to denote like features throughout the specification andfigures.

SUMMARY

Various implementations disclosed herein include devices, systems, andmethods for presenting synthesized reality (SR) content in associatedwith recognized objects. According to some implementations, the methodis performed at a device including non-transitory memory and one or moreprocessors coupled with the non-transitory memory. The method includes:obtaining image data from an image sensor; recognizing a portion of anobject within the image data; obtaining SR content associated with theportion of the object; and displaying the SR content in association withthe portion of the object.

In accordance with some implementations, a device includes one or moreprocessors, a non-transitory memory, and one or more programs; the oneor more programs are stored in the non-transitory memory and configuredto be executed by the one or more processors and the one or moreprograms include instructions for performing or causing performance ofany of the methods described herein. In accordance with someimplementations, a non-transitory computer readable storage medium hasstored therein instructions, which, when executed by one or moreprocessors of a device, cause the device to perform or cause performanceof any of the methods described herein. In accordance with someimplementations, a device includes: one or more processors, anon-transitory memory, and means for performing or causing performanceof any of the methods described herein.

DESCRIPTION

Numerous details are described in order to provide a thoroughunderstanding of the example implementations shown in the drawings.However, the drawings merely show some example aspects of the presentdisclosure and are therefore not to be considered limiting. Those ofordinary skill in the art will appreciate that other effective aspectsand/or variants do not include all of the specific details describedherein. Moreover, well-known systems, methods, components, devices andcircuits have not been described in exhaustive detail so as not toobscure more pertinent aspects of the example implementations describedherein.

A physical setting refers to a world that individuals can sense and/orwith which individuals can interact without assistance of electronicsystems. Physical settings (e.g., a physical forest) include physicalelements (e.g., physical trees, physical structures, and physicalanimals). Individuals can directly interact with and/or sense thephysical setting, such as through touch, sight, smell, hearing, andtaste.

In contrast, a synthesized reality (SR) setting refers to an entirely orpartly computer-created setting that individuals can sense and/or withwhich individuals can interact via an electronic system. In SR, a subsetof an individual's movements is monitored, and, responsive thereto, oneor more attributes of one or more virtual objects in the SR setting ischanged in a manner that conforms with one or more physical laws. Forexample, a SR system may detect an individual walking a few pacesforward and, responsive thereto, adjust graphics and audio presented tothe individual in a manner similar to how such scenery and sounds wouldchange in a physical setting. Modifications to attribute(s) of virtualobject(s) in a SR setting also may be made responsive to representationsof movement (e.g., audio instructions).

An individual may interact with and/or sense a SR object using any oneof his senses, including touch, smell, sight, taste, and sound. Forexample, an individual may interact with and/or sense aural objects thatcreate a multi-dimensional (e.g., three dimensional) or spatial auralsetting, and/or enable aural transparency. Multi-dimensional or spatialaural settings provide an individual with a perception of discrete auralsources in multi-dimensional space. Aural transparency selectivelyincorporates sounds from the physical setting, either with or withoutcomputer-created audio. In some SR settings, an individual may interactwith and/or sense only aural objects.

One example of SR is virtual reality (VR). A VR setting refers to asimulated setting that is designed only to include computer-createdsensory inputs for at least one of the senses. A VR setting includesmultiple virtual objects with which an individual may interact and/orsense. An individual may interact and/or sense virtual objects in the VRsetting through a simulation of a subset of the individual's actionswithin the computer-created setting, and/or through a simulation of theindividual or his presence within the computer-created setting.

Another example of SR is mixed reality (MR). A MR setting refers to asimulated setting that is designed to integrate computer-created sensoryinputs (e.g., virtual objects) with sensory inputs from the physicalsetting, or a representation thereof. On a reality spectrum, a mixedreality setting is between, and does not include, a VR setting at oneend and an entirely physical setting at the other end.

In some MR settings, computer-created sensory inputs may adapt tochanges in sensory inputs from the physical setting. Also, someelectronic systems for presenting MR settings may monitor orientationand/or location with respect to the physical setting to enableinteraction between virtual objects and real objects (which are physicalelements from the physical setting or representations thereof). Forexample, a system may monitor movements so that a virtual plant appearsstationery with respect to a physical building.

One example of mixed reality is augmented reality (AR). An AR settingrefers to a simulated setting in which at least one virtual object issuperimposed over a physical setting, or a representation thereof. Forexample, an electronic system may have an opaque display and at leastone imaging sensor for capturing images or video of the physicalsetting, which are representations of the physical setting. The systemcombines the images or video with virtual objects, and displays thecombination on the opaque display. An individual, using the system,views the physical setting indirectly via the images or video of thephysical setting, and observes the virtual objects superimposed over thephysical setting. When a system uses image sensor(s) to capture imagesof the physical setting, and presents the AR setting on the opaquedisplay using those images, the displayed images are called a videopass-through. Alternatively, an electronic system for displaying an ARsetting may have a transparent or semi-transparent display through whichan individual may view the physical setting directly. The system maydisplay virtual objects on the transparent or semi-transparent display,so that an individual, using the system, observes the virtual objectssuperimposed over the physical setting. In another example, a system maycomprise a projection system that projects virtual objects into thephysical setting. The virtual objects may be projected, for example, ona physical surface or as a holograph, so that an individual, using thesystem, observes the virtual objects superimposed over the physicalsetting.

An augmented reality setting also may refer to a simulated setting inwhich a representation of a physical setting is altered bycomputer-created sensory information. For example, a portion of arepresentation of a physical setting may be graphically altered (e.g.,enlarged), such that the altered portion may still be representative ofbut not a faithfully-reproduced version of the originally capturedimage(s). As another example, in providing video pass-through, a systemmay alter at least one of the sensor images to impose a particularviewpoint different than the viewpoint captured by the image sensor(s).As an additional example, a representation of a physical setting may bealtered by graphically obscuring or excluding portions thereof.

Another example of mixed reality is augmented virtuality (AV). An AVsetting refers to a simulated setting in which a computer-created orvirtual setting incorporates at least one sensory input from thephysical setting. The sensory input(s) from the physical setting may berepresentations of at least one characteristic of the physical setting.For example, a virtual object may assume a color of a physical elementcaptured by imaging sensor(s). In another example, a virtual object mayexhibit characteristics consistent with actual weather conditions in thephysical setting, as identified via imaging, weather-related sensors,and/or online weather data. In yet another example, an augmented realityforest may have virtual trees and structures, but the animals may havefeatures that are accurately reproduced from images taken of physicalanimals.

Many electronic systems enable an individual to interact with and/orsense various SR settings. One example includes head mounted systems. Ahead mounted system may have an opaque display and speaker(s).Alternatively, a head mounted system may be designed to receive anexternal display (e.g., a smartphone). The head mounted system may haveimaging sensor(s) and/or microphones for taking images/video and/orcapturing audio of the physical setting, respectively. A head mountedsystem also may have a transparent or semi-transparent display. Thetransparent or semi-transparent display may incorporate a substratethrough which light representative of images is directed to anindividual's eyes. The display may incorporate LEDs, OLEDs, a digitallight projector, a laser scanning light source, liquid crystal onsilicon, or any combination of these technologies. The substrate throughwhich the light is transmitted may be a light waveguide, opticalcombiner, optical reflector, holographic substrate, or any combinationof these substrates. In one embodiment, the transparent orsemi-transparent display may transition selectively between an opaquestate and a transparent or semi-transparent state. In another example,the electronic system may be a projection-based system. Aprojection-based system may use retinal projection to project imagesonto an individual's retina. Alternatively, a projection system also mayproject virtual objects into a physical setting (e.g., onto a physicalsurface or as a holograph). Other examples of SR systems include headsup displays, automotive windshields with the ability to displaygraphics, windows with the ability to display graphics, lenses with theability to display graphics, headphones or earphones, speakerarrangements, input mechanisms (e.g., controllers having or not havinghaptic feedback), tablets, smartphones, and desktop or laptop computers.

Today a user is able to drag-and-drop synthesized reality (SR) contentonto a user interface so that the SR content is overlaid on his/herphysical setting. However, the SR content is often selected by the userfrom a library of existing SR content. In contrast, a user may wish toview SR content that is context sensitive (e.g., based on objectsrecognized in the physical setting). As such, the disclosedimplementations detect an object within a physical setting and presentSR on or around the object, wherein the SR content is associated withthe object.

To this end, SR content may be superimposed on an object in order toprovide a cutaway or “x-ray” view into the object. In someimplementations, the SR content is obtained from a library of SR contentassociated with the object (e.g., a cutaway or exploded view of anobject, video content associated with the object, or the like). As such,for example, the user is able to see SR content associated with theobject superimposed on the object. In some implementations, the SRcontent is generated based on sensor data associated with the object(e.g., sensors integrated with or embedded within the object such asautomobile sensors) and/or environmental sensors (e.g., electromagneticwaves penetrating a building or wind gusts striking a tree or building).As such, for example, the user is able to see live SR content associatedwith the object or the setting around the object. In someimplementations, the SR content is selected based on the user's ordevices' orientation/location relative to the object (e.g., device orcamera pose). As such, for example, the user may see different SRcontent associated with the object when looking at an object from atop-down view as opposed to a perspective view.

FIG. 1A is a block diagram of an example operating architecture 100A inaccordance with some implementations. While pertinent features areshown, those of ordinary skill in the art will appreciate from thepresent disclosure that various other features have not been illustratedfor the sake of brevity and so as not to obscure more pertinent aspectsof the example implementations disclosed herein. To that end, as anon-limiting example, the operating architecture 100A includes anelectronic device 120 and an optional display device 130.

In some implementations, the electronic device 120 is configured topresent the SR experience to a user. In some implementations, theelectronic device 120 includes a suitable combination of software,firmware, and/or hardware. The electronic device 120 is described ingreater detail below with respect to FIG. 3. According to someimplementations, the electronic device 120 presents a synthesizedreality (SR) experience to the user while the user is physically presentwithin a physical setting 105 that includes an object 160 (e.g., aphysical model, toy, or other volumetric item) on a table 155 within thefield-of-view 111 of the electronic device 120. As such, in someimplementations, the user holds the electronic device 120 in his/herhand(s). In some implementations, while presenting an augmented reality(AR) experience, the electronic device 120 is configured to present ARcontent (e.g., an AR cylinder 109) and to enable video pass-through ofthe physical setting 105 (e.g., including the object 160 and the table155) on a display 122.

In some implementations, the display device 130 is configured to presentmedia content (e.g., video and/or audio content) to the user. In someimplementations, the display device 130 corresponds to a television or acomputing device such as a desktop computer, kiosk, laptop computer,tablet, mobile phone, wearable computing device, or the like. In someimplementations, the display device 130 includes a suitable combinationof software, firmware, and/or hardware. The display device 130 isdescribed in greater detail below with respect to FIG. 4.

FIG. 1B is a block diagram of an example operating architecture 100B inaccordance with some implementations. While pertinent features areshown, those of ordinary skill in the art will appreciate from thepresent disclosure that various other features have not been illustratedfor the sake of brevity and so as not to obscure more pertinent aspectsof the example implementations disclosed herein. To that end, as anon-limiting example, the operating architecture 100B includes acontroller 110, an electronic device 120, and an optional display device130.

In some implementations, the controller 110 is configured to manage andcoordinate an SR experience for the user. In some implementations, thecontroller 110 includes a suitable combination of software, firmware,and/or hardware. The controller 110 is described in greater detail belowwith respect to FIG. 2. In some implementations, the controller 110 is acomputing device that is local or remote relative to the physicalsetting 105. For example, the controller 110 is a local server locatedwithin the physical setting 105. In another example, the controller 110is a remote server located outside of the physical setting 105 (e.g., acloud server, central server, etc.).

In some implementations, the controller 110 is communicatively coupledwith the electronic device 120 via one or more wired or wirelesscommunication channels 144 (e.g., BLUETOOTH, IEEE 802.11x, IEEE 802.16x,IEEE 802.3x, etc.). In some implementations, the controller 110 iscommunicatively coupled with the display device 130 via one or morewired or wireless communication channels 142 (e.g., BLUETOOTH, IEEE802.11x, IEEE 802.16x, IEEE 802.3x, etc.). In some implementations, theelectronic device 120 is communicatively coupled with the display device130 via one or more wired or wireless communication channels 146 (e.g.,BLUETOOTH, IEEE 802.11x, IEEE 802.16x, IEEE 802.3x, etc.).

In some implementations, the electronic device 120 is configured topresent the SR experience to the user 150. In some implementations, theelectronic device 120 includes a suitable combination of software,firmware, and/or hardware. The electronic device 120 is described ingreater detail below with respect to FIG. 3. In some implementations,the functionalities of the controller 110 and/or the display device 130are provided by and/or combined with the electronic device 120.

According to some implementations, the electronic device 120 presents asynthesized reality (SR) experience to the user 150 while the user 150is virtually and/or physically present within a physical setting 105that includes an object 160 (e.g., a physical model, toy, or othervolumetric item) on a table 155. In some implementations, whilepresenting an augmented reality (AR) experience, the electronic device120 is configured to present AR content and to enable opticalsee-through of the physical setting 105. In some implementations, whilepresenting a virtual reality (VR) experience, the electronic device 120is configured to present VR content and to optionally enable videopass-through of the physical setting 105.

In some implementations, the user 150 wears the electronic device 120 onhis/her head such as a head-mounted device (HMD). As such, theelectronic device 120 includes one or more displays provided to displaythe SR content. For example, the electronic device 120 encloses thefield-of-view of the user 150. As another example, the electronic device120 slides into or otherwise attaches to a head mounted enclosure. Insome implementations, the electronic device 120 is replaced with an SRchamber, enclosure, or room configured to present SR content in whichthe user 150 does not wear the electronic device 120.

In some implementations, the display device 130 is configured to presentmedia content (e.g., video and/or audio content) to the user 150. Insome implementations, the display device 130 corresponds to a televisionor a computing device such as a desktop computer, kiosk, laptopcomputer, tablet, mobile phone, wearable computing device, or the like.In some implementations, the display device 130 includes a suitablecombination of software, firmware, and/or hardware. The display device130 is described in greater detail below with respect to FIG. 4.

FIG. 2 is a block diagram of an example of the controller 110 inaccordance with some implementations. While certain specific featuresare illustrated, those skilled in the art will appreciate from thepresent disclosure that various other features have not been illustratedfor the sake of brevity, and so as not to obscure more pertinent aspectsof the implementations disclosed herein. To that end, as a non-limitingexample, in some implementations, the controller 110 includes one ormore processing units 202 (e.g., microprocessors, application-specificintegrated-circuits (ASICs), field-programmable gate arrays (FPGAs),graphics processing units (GPUs), central processing units (CPUs),processing cores, and/or the like), one or more input/output (I/O)devices 206, one or more communication interfaces 208 (e.g., universalserial bus (USB), IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, global systemfor mobile communications (GSM), code division multiple access (CDMA),time division multiple access (TDMA), global positioning system (GPS),infrared (IR), BLUETOOTH, ZIGBEE, and/or the like type interface), oneor more programming (e.g., I/O) interfaces 210, a memory 220, and one ormore communication buses 204 for interconnecting these and various othercomponents.

In some implementations, the one or more communication buses 204 includecircuitry that interconnects and controls communications between systemcomponents. In some implementations, the one or more I/O devices 206include at least one of a keyboard, a mouse, a touchpad, a joystick, oneor more microphones, one or more speakers, one or more image sensors,one or more displays, and/or the like.

The memory 220 includes high-speed random-access memory, such as dynamicrandom-access memory (DRAM), static random-access memory (SRAM),double-data-rate random-access memory (DDR RAM), or other random-accesssolid-state memory devices. In some implementations, the memory 220includes non-volatile memory, such as one or more magnetic disk storagedevices, optical disk storage devices, flash memory devices, or othernon-volatile solid-state storage devices. The memory 220 optionallyincludes one or more storage devices remotely located from the one ormore processing units 202. The memory 220 comprises a non-transitorycomputer readable storage medium. In some implementations, the memory220 or the non-transitory computer readable storage medium of the memory220 stores the following programs, modules and data structures, or asubset thereof including an optional operating system 230 and asynthesized reality (SR) experience engine 240.

The operating system 230 includes procedures for handling various basicsystem services and for performing hardware dependent tasks. In someimplementations, the SR experience engine 240 is configured to manageand coordinate one or more SR experiences for one or more users (e.g., asingle SR experience for one or more users, or multiple SR experiencesfor respective groups of one or more users). To that end, in variousimplementations, the SR experience engine 240 includes a data obtainer242, an object recognizer 243, a mapper and locator engine 244, an SRcontent obtainer 246, an SR content manager 248, and a data transmitter250.

In some implementations, the data obtainer 242 is configured to obtaindata (e.g., presentation data, user interaction data, sensor data,location data, etc.) from at least one of sensors in the physicalsetting 105, sensors associated with the controller 110, the electronicdevice 120, and the display device 130. For example, the data obtainer242 obtains sensor data from the electronic device 120 that includesimage data from external facing image sensors of the electronic device120, wherein the image data corresponds to images or a video streamcapturing the physical setting 105. To that end, in variousimplementations, the data obtainer 242 includes instructions and/orlogic therefor, and heuristics and metadata therefor.

In some implementations, the object recognizer 243 is configured torecognize objects within the physical setting 105 based on image datafrom the electronic device 120 or other image sensors within thephysical setting 105. To that end, in various implementations, theobject recognizer 243 includes instructions and/or logic therefor, andheuristics and metadata therefor.

In some implementations, the mapper and locator engine 244 is configuredto map the physical setting 105 and to track the position/location ofthe electronic device 120 or the user 150 with respect to the physicalsetting 105. In some implementations, the mapper and locator engine 244is also configured to determine the orientation of the electronic device120 or the user 150 (e.g., device or camera pose) relative to one ormore reference points (e.g., the object 160) in the physical setting(e.g., the center of mass of the object 160 or another point on theobject 160). According to some implementations, the mapper and locatorengine 244 determines the orientation of the electronic device 120relative to the object based on the techniques described in U.S.Provisional Patent Application No. 62/556,849, Attorney Docket No.173PR, filed Sep. 11, 2017, which is incorporated herein in itsentirety. To that end, in various implementations, the mapper andlocator engine 244 includes instructions and/or logic therefor, andheuristics and metadata therefor.

In some implementations, the SR content obtainer 246 is configured toobtain (e.g., receive, retrieve, or generate) SR content associated withobjects recognized within the physical setting 105. To that end, invarious implementations, the SR content obtainer 246 includesinstructions and/or logic therefor, and heuristics and metadatatherefor.

In some implementations, the SR content manager 248 is configured tomanage and coordinate the presentation of the SR in association with anobject recognized within the physical setting 105 as the orientation ofthe user or electronic device 120 changes relative to the recognizedobject 160. To that end, in various implementations, the SR contentmanager 248 includes instructions and/or logic therefor, and heuristicsand metadata therefor.

In some implementations, the data transmitter 250 is configured totransmit data (e.g., presentation data, location data, etc.) to at leastone of the electronic device 120 and the display device 130. To thatend, in various implementations, the data transmitter 250 includesinstructions and/or logic therefor, and heuristics and metadatatherefor.

Although the data obtainer 242, the object recognizer 243, the mapperand locator engine 244, the SR content obtainer 246, the SR contentmanager 248, and the data transmitter 250 are shown as residing on asingle device (e.g., the controller 110), it should be understood thatin other implementations, any combination of the data obtainer 242, theobject recognizer 243, the mapper and locator engine 244, the SR contentobtainer 246, the SR content manager 248, and the data transmitter 250may be located in separate computing devices.

Moreover, FIG. 2 is intended more as a functional description of thevarious features which are present in a particular embodiment as opposedto a structural schematic of the implementations described herein. Asrecognized by those of ordinary skill in the art, items shown separatelycould be combined and some items could be separated. For example, somefunctional modules shown separately in FIG. 2 could be implemented in asingle module and the various functions of single functional blockscould be implemented by one or more functional blocks in variousimplementations. The actual number of modules and the division ofparticular functions and how features are allocated among them will varyfrom one embodiment to another and, in some implementations, depends inpart on the particular combination of hardware, software, and/orfirmware chosen for a particular embodiment.

FIG. 3 is a block diagram of an example of the electronic device 120(e.g., an HMD, mobile phone, or tablet) in accordance with someimplementations. While certain specific features are illustrated, thoseskilled in the art will appreciate from the present disclosure thatvarious other features have not been illustrated for the sake ofbrevity, and so as not to obscure more pertinent aspects of theimplementations disclosed herein. To that end, as a non-limitingexample, in some implementations, the electronic device 120 includes oneor more processing units 302 (e.g., microprocessors, ASICs, FPGAs, GPUs,CPUs, processing cores, and/or the like), one or more input/output (I/O)devices and sensors 306, one or more communication interfaces 308 (e.g.,USB, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, GSM, CDMA, TDMA, GPS, IR,BLUETOOTH, ZIGBEE, and/or the like type interface), one or moreprogramming (e.g., I/O) interfaces 310, one or more displays 312, one ormore optional interior and/or exterior facing image sensors 314, one ormore optional depth sensors 316, a memory 320, and one or morecommunication buses 304 for interconnecting these and various othercomponents.

In some implementations, the one or more communication buses 304 includecircuitry that interconnects and controls communications between systemcomponents. In some implementations, the one or more I/O devices andsensors 306 include at least one of an inertial measurement unit (IMU),an accelerometer, a gyroscope, a thermometer, one or more physiologicalsensors (e.g., blood pressure monitor, heart rate monitor, blood oxygensensor, blood glucose sensor, etc.), one or more microphones, one ormore speakers, a haptics engine, a heating and/or cooling unit, a skinshear engine, and/or the like.

In some implementations, the one or more displays 312 are configured topresent the SR experience to the user. In some implementations, the oneor more displays 312 are also configured to present flat video contentto the user (e.g., a 2-dimensional or “flat” AVI, FLV, WMV, MOV, MP4, orthe like file associated with a TV episode or a movie, or live videopass-through of the physical setting 105). In some implementations, theone or more displays 312 correspond to holographic, digital lightprocessing (DLP), liquid-crystal display (LCD), liquid-crystal onsilicon (LCoS), organic light-emitting field-effect transitory (OLET),organic light-emitting diode (OLED), surface-conduction electron-emitterdisplay (SED), field-emission display (FED), quantum-dot light-emittingdiode (QD-LED), micro-electro-mechanical system (MEMS), and/or the likedisplay types. In some implementations, the one or more displays 312correspond to diffractive, reflective, polarized, holographic, etc.waveguide displays. For example, the electronic device 120 includes asingle display. In another example, the electronic device 120 includes adisplay for each eye of the user. In some implementations, the one ormore displays 312 are capable of presenting AR and VR content. In someimplementations, the one or more displays 312 are capable of presentingAR or VR content.

In some implementations, the one or more optional image sensors 314 areconfigured to obtain image data that corresponds to at least a portionof the face of the user that includes the eyes of the user. For example,the one or more optional image sensors 314 correspond to one or more RGBcameras (e.g., with a complementary metal-oxide-semiconductor (CMOS)image sensor or a charge-coupled device (CCD) image sensor), infrared(IR) image sensors, event-based cameras, and/or the like.

In some implementations, the one or more optional depth sensors 316 areconfigured to obtain depth data that corresponds to at least a portionof the face of the user and to synthesize a depth/mesh map of the faceof the user, where the mesh map characterizes the facial topography ofthe user. For example, the one or more optional depth sensors 316correspond to a structured light device, a time-of-flight device, and/orthe like.

The memory 320 includes high-speed random-access memory, such as DRAM,SRAM, DDR RAM, or other random-access solid-state memory devices. Insome implementations, the memory 320 includes non-volatile memory, suchas one or more magnetic disk storage devices, optical disk storagedevices, flash memory devices, or other non-volatile solid-state storagedevices. The memory 320 optionally includes one or more storage devicesremotely located from the one or more processing units 302. The memory320 comprises a non-transitory computer readable storage medium. In someimplementations, the memory 320 or the non-transitory computer readablestorage medium of the memory 320 stores the following programs, modulesand data structures, or a subset thereof including an optional operatingsystem 330 and an SR presentation engine 340.

The operating system 330 includes procedures for handling various basicsystem services and for performing hardware dependent tasks. In someimplementations, the SR presentation engine 340 is configured to presentSR content to the user via the one or more displays 312. To that end, invarious implementations, the SR presentation engine 340 includes anoptional object recognizer 341, a data obtainer 342, an optionalorientation determiner 343, an SR presenter 344, a user interactionhandler 346, and a data transmitter 350.

In some implementations, the optional object recognizer 341 isconfigured to recognize objects within the physical setting 105 based onimage data from the one or more image sensors 314 (e.g., external facingimage sensors). To that end, in various implementations, the objectrecognizer 341 includes instructions and/or logic therefor, andheuristics and metadata therefor.

In some implementations, the data obtainer 342 is configured to obtaindata (e.g., presentation data, user interaction data, sensor data,location data, etc.) from at least one of sensors in the physicalsetting 105, sensors associated with the electronic device 120, thecontroller 110, and the display device 130. To that end, in variousimplementations, the data obtainer 342 includes instructions and/orlogic therefor, and heuristics and metadata therefor.

In some implementations, the optional orientation determiner 343 isconfigured to determine the orientation of the electronic device 120 orthe user 150 relative to one or more reference points (e.g., the objectobject) in the physical setting (e.g., the center of mass of the object160 or another point on the object 160). For example, in someimplementations, the orientation determiner 343 determines theorientation of the electronic device 120 relative to the object based onthe techniques described in U.S. Provisional Patent Application No.62/556,849, Attorney Docket No. 173PR, filed Sep. 11, 2017, which isincorporated herein in its entirety. To that end, in variousimplementations, the orientation determiner 343 includes instructionsand/or logic therefor, and heuristics and metadata therefor.

In some implementations, the SR presenter 344 is configured to presentSR content via the one or more displays 312. In some implementations,the SR presenter 344 is also configured to present flat video contentvia the one or more displays 312. To that end, in variousimplementations, the SR presenter 344 includes instructions and/or logictherefor, and heuristics and metadata therefor.

In some implementations, the user interaction handler 346 is configuredto detect and interpret user interactions with the presented SR content.To that end, in various implementations, the user interaction handler346 includes instructions and/or logic therefor, and heuristics andmetadata therefor.

In some implementations, the data transmitter 350 is configured totransmit data (e.g., presentation data, location data, user interactiondata, etc.) to at least one of the controller 110 and the display device130. To that end, in various implementations, the data transmitter 350includes instructions and/or logic therefor, and heuristics and metadatatherefor.

Although the optional object recognizer 341, the data obtainer 342, theoptional orientation determiner 343, the SR presenter 344, the userinteraction handler 346, and the data transmitter 350 are shown asresiding on a single device (e.g., the electronic device 120), it shouldbe understood that in other implementations, any combination of theoptional object recognizer 341, the data obtainer 342, the optionalorientation determiner 343, the SR presenter 344, the user interactionhandler 346, and the data transmitter 350 may be located in separatecomputing devices.

Moreover, FIG. 3 is intended more as a functional description of thevarious features which are present in a particular embodiment as opposedto a structural schematic of the implementations described herein. Asrecognized by those of ordinary skill in the art, items shown separatelycould be combined and some items could be separated. For example, somefunctional modules shown separately in FIG. 3 could be implemented in asingle module and the various functions of single functional blockscould be implemented by one or more functional blocks in variousimplementations. The actual number of modules and the division ofparticular functions and how features are allocated among them will varyfrom one embodiment to another and, in some implementations, depends inpart on the particular combination of hardware, software, and/orfirmware chosen for a particular embodiment.

FIG. 4 is a block diagram of an example of the optional display device130 (e.g., a television (TV) or other display within the physicalsetting 105) in accordance with some implementations. While certainspecific features are illustrated, those skilled in the art willappreciate from the present disclosure that various other features havenot been illustrated for the sake of brevity, and so as not to obscuremore pertinent aspects of the implementations disclosed herein. To thatend, as a non-limiting example, in some implementations the displaydevice 130 includes one or more processing units 402 (e.g.,microprocessors, ASICs, FPGAs, GPUs, CPUs, processing cores, and/or thelike), one or more input/output (I/O) devices and sensors 406, one ormore communication interfaces 408 (e.g., USB, IEEE 802.3x, IEEE 802.11x,IEEE 802.16x, GSM, CDMA, TDMA, GPS, IR, BLUETOOTH, ZIGBEE, and/or thelike type interface), one or more programming (e.g., I/O) interfaces410, a display 412, a memory 420, and one or more communication buses404 for interconnecting these and various other components. In someimplementations, the display device 130 is optionally controlled by aremote-control device, voice commands, the electronic device 120, or thelike.

In some implementations, the one or more communication buses 404 includecircuitry that interconnects and controls communications between systemcomponents. In some implementations, the one or more I/O devices andsensors 406 include at least one of one or more IR sensors, one or morephysical buttons, one or more microphones, one or more speakers, one ormore image sensors, one or more depth sensors, and/or the like.

In some implementations, the display 412 corresponds to holographic,digital light processing (DLP), liquid-crystal display (LCD),liquid-crystal on silicon (LCoS), organic light-emitting field-effecttransitory (OLET), organic light-emitting diode (OLED),surface-conduction electron-emitter display (SED), field-emissiondisplay (FED), quantum-dot light-emitting diode (QD-LED),micro-electro-mechanical system (MEMS), and/or the like display types.

The memory 420 includes high-speed random-access memory, such as DRAM,SRAM, DDR RAM, or other random-access solid-state memory devices. Insome implementations, the memory 420 includes non-volatile memory, suchas one or more magnetic disk storage devices, optical disk storagedevices, flash memory devices, or other non-volatile solid-state storagedevices. The memory 420 optionally includes one or more storage devicesremotely located from the one or more processing units 402. The memory420 comprises a non-transitory computer readable storage medium. In someimplementations, the memory 420 or the non-transitory computer readablestorage medium of the memory 420 stores the following programs, modulesand data structures, or a subset thereof including an optional operatingsystem 430 and a presentation engine 440.

The operating system 430 includes procedures for handling various basicsystem services and for performing hardware dependent tasks. In someimplementations, the presentation engine 440 is configured to presentmedia content (e.g., video and/or audio content) to users via thedisplay 412 and the one or more I/O devices and sensors 406 (e.g., oneor more speakers). To that end, in various implementations, thepresentation engine 440 includes a data obtainer 442, a contentpresenter 444, an interaction handler 446, and a data transmitter 450.

In some implementations, the data obtainer 442 is configured to obtaindata (e.g., presentation data, user interaction data, etc.) from atleast one of sensors in the physical setting 105, sensors associatedwith the display device 130, the controller 110, and the electronicdevice 120. To that end, in various implementations, the data obtainer442 includes instructions and/or logic therefor, and heuristics andmetadata therefor.

In some implementations, the content presenter 444 is configured torender and display video content via the display 412. To that end, invarious implementations, the content presenter 444 includes instructionsand/or logic therefor, and heuristics and metadata therefor.

In some implementations, the interaction handler 446 is configured todetect and interpret user interactions with the display device 130(e.g., navigation, playback, tuning, volume adjustment, or the likecommands). To that end, in various implementations, the interactionhandler 446 includes instructions and/or logic therefor, and heuristicsand metadata therefor.

In some implementations, the data transmitter 450 is configured totransmit data (e.g., presentation data, user interaction data, etc.) toat least one of the controller 110 and the electronic device 120. Tothat end, in various implementations, the data transmitter 450 includesinstructions and/or logic therefor, and heuristics and metadatatherefor.

Although the data obtainer 442, the content presenter 444, theinteraction handler 446, and the data transmitter 450 are shown asresiding on a single device (e.g., the display device 130), it should beunderstood that in other implementations, any combination of the dataobtainer 442, the content presenter 444, the interaction handler 446,and the data transmitter 450 may be located in separate computingdevices.

Moreover, FIG. 4 is intended more as a functional description of thevarious features which are present in a particular embodiment as opposedto a structural schematic of the implementations described herein. Asrecognized by those of ordinary skill in the art, items shown separatelycould be combined and some items could be separated. For example, somefunctional modules shown separately in FIG. 4 could be implemented in asingle module and the various functions of single functional blockscould be implemented by one or more functional blocks in variousimplementations. The actual number of modules and the division ofparticular functions and how features are allocated among them will varyfrom one embodiment to another and, in some implementations, depends inpart on the particular combination of hardware, software, and/orfirmware chosen for a particular embodiment.

FIGS. 5A-5B illustrate example SR presentation scenarios 500 and 550 inaccordance with some implementations. While pertinent features areshown, those of ordinary skill in the art will appreciate from thepresent disclosure that various other features have not been illustratedfor the sake of brevity and so as not to obscure more pertinent aspectsof the example implementations disclosed herein.

FIG. 5A illustrates an example SR presentation scenario 500. As shown inFIG. 5A, the physical setting 505 includes a model submarine 512 sittingon a table 514. In this example, a portion of the physical setting 505is within the field-of-view 520 of the electronic device 120, where thefield-of-view 520 is associated with an external facing image sensor ofthe electronic device 120 (e.g., a tablet or mobile phone). In otherwords, the user is looking at the model submarine 412 from a side orperspective orientation through the electronic device 120. As such, theportion of the physical setting 505, including the model submarine 512sitting on the table 514, is displayed on the display 510 of theelectronic device 120 (e.g., a live video stream or video pass-throughof the physical setting 505). As shown in FIG. 5A, the electronic device120 superimposes AR content 525 on the model submarine 512 according tothe side or perspective orientation/view of the electronic device 120relative to the model submarine 512. For example, the AR content 525corresponds to an AR cutaway view of the model submarine 512 showing thevarious decks of the type of submarine associated with the modelsubmarine 512 (e.g., a Russian Typhoon class).

FIG. 5B illustrates an example SR presentation scenario 550. As shown inFIG. 5B, the physical setting 505 includes a model submarine 512 sittingon a table 514. In this example, a portion of the physical setting 505is within the field-of-view 520 of the electronic device 120 (not showndue to the top-down viewing angle), where the field-of-view 520 isassociated with an external facing image sensor of the electronic device120 (e.g., a tablet or mobile phone). In other words, the user islooking at the model submarine 412 from a top-down orientation throughthe electronic device 120. As such, the portion of the physical setting505, including the model submarine 512 sitting on the table 514, isdisplayed on the display 510 of the electronic device 120 (e.g., a livevideo stream or video pass-through of the physical setting 505). Asshown in FIG. 5B, the electronic device 120 superimposes AR content 575on the model submarine 512 according to the top-down orientation/view ofthe electronic device 120 relative to the model submarine 512. Forexample, the AR content 575 corresponds to an AR cutaway view of themodel submarine 512 showing vertical missile launch tubes associatedwith the model submarine 512.

According to some implementations, as shown in FIGS. 5A-5B, the ARcontent changes based on the orientation of the electronic device 120relative to the object. In FIG. 5A, the electronic device 120superimposes the AR content 525 on the model submarine 512 based on theside or perspective orientation of the electronic device 120 relative tothe model submarine 512. In contrast, in FIG. 5B, the electronic device120 superimposes the AR content 575 on the model submarine 512 based onthe top-down orientation of the electronic device 120 relative to themodel submarine 512. In some implementations, the AR content 525 and theAR content 575 correspond to the same AR content from differentpoints-of-view. In some implementations, the AR content 525 and the ARcontent 575 correspond to different AR content based on thepoint-of-view of the electronic device 120.

FIGS. 6A-6B illustrate example SR presentation scenarios 600 and 650 inaccordance with some implementations. While pertinent features areshown, those of ordinary skill in the art will appreciate from thepresent disclosure that various other features have not been illustratedfor the sake of brevity and so as not to obscure more pertinent aspectsof the example implementations disclosed herein.

FIG. 6A illustrates an example SR presentation scenario 600. As shown inFIG. 6A, the physical setting 605 includes the Eiffel Tower 612. In thisexample, a portion of the physical setting 605 is within thefield-of-view 620 of the electronic device 120, where the field-of-view620 is associated with an external facing image sensor of the electronicdevice 120 (e.g., a tablet or mobile phone). In other words, the user islooking at a portion of the Eiffel Tower 612 (e.g., the lower portion ofthe Eiffel Tower 612 below the first platform) from a side orperspective orientation through the electronic device 120. As such, theportion of the physical setting 605, including the lower portion of theEiffel Tower 612, is displayed on the display 610 of the electronicdevice 120 (e.g., a live video stream or video pass-through of thephysical setting 605).

According to some implementations, a user is able to see AR contentsuperimposed on an object that is generated based on sensor dataassociated with the object and/or environmental sensors proximate to theobject. As shown in FIG. 6A, the electronic device 120 displays ARcontent 625 a, 625 b, and 625 c. For example, the AR content 625 asuperimposed on the Eiffel Tower 612 corresponds to a first elevatormoving from the ground toward the first platform based on sensor datafrom the Eiffel Tower 612. For example, the AR content 625 b correspondsto the crowd of people below the Eiffel Tower 612 or environmentalstatistics (e.g., wind speed, temperature, humidity, etc.) based onsensor data from sensors associated with the electronic device 120,sensors associated with the Eiffel Tower 612, or sensors within thephysical setting 605. For example, the AR content 625 c superimposed onthe Eiffel Tower 612 corresponds to a second elevator moving from thefirst platform toward the ground based on sensor data from the EiffelTower 612.

FIG. 6B illustrates an example SR presentation scenario 650. As shown inFIG. 6B, the physical setting 655 includes an automobile 662. In thisexample, a portion of the physical setting 655 is within thefield-of-view 620 of the electronic device 120, where the field-of-view620 is associated with an external facing image sensor of the electronicdevice 120 (e.g., a tablet or mobile phone). In other words, the user islooking at a portion of the automobile 662 from a front orientationthrough the electronic device 120 (e.g., the engine compartment portionof the automobile 662). As such, the portion of the physical setting655, including the engine compartment portion of the automobile 662, isdisplayed on the display 610 of the electronic device 120 (e.g., a livevideo stream or video pass-through of the physical setting 655).

According to some implementations, a user is able to see AR contentsuperimposed on an object that is generated based on sensor data fromthe object. As shown in FIG. 6B, the electronic device 120 displays ARcontent 675 a and 675 b superimposed on the engine compartment portionof the automobile 662. For example, the AR content 675 a superimposed onthe engine compartment portion of the automobile 662 corresponds to anillustration of the working transmission of the automobile 662 based onsensor data from the automobile 662. For example, the AR content 675 bsuperimposed on the engine compartment portion of the automobile 662corresponds to an illustration of the working engine of the automobile662 based on sensor data from the automobile 662.

FIG. 7 illustrates an example SR presentation scenario 700 in accordancewith some implementations. While pertinent features are shown, those ofordinary skill in the art will appreciate from the present disclosurethat various other features have not been illustrated for the sake ofbrevity and so as not to obscure more pertinent aspects of the exampleimplementations disclosed herein.

According to some implementations, a user is able to seamlessly switchbetween viewing a movie on a display device and an SR reconstructionthereof superimposed on a related physical object or model scene. Asshown in FIG. 7, the physical setting 105 includes the display device130, the user 150, and the model submarine 160 sitting on the table 155.In state 710 (e.g., time T), the user 150 is watching video content 705(e.g., a television (TV) show or movie associated with a submarine crew)on the display device 130. In state 720 (e.g., time T+1), the user 150is wearing the electronic device 120 (e.g., an HMD) on his/her head andviewing an SR reconstruction 715 of the video content 705 beingpresented on and/or around the model submarine 160. As such, in state720, the user 150 is able to see video pass-through of the physicalsetting within his/her field-of-view 105 and also is being presented theSR reconstruction 715 of the video content 705 superimposed on and/oraround the model submarine 612.

FIG. 8 is a flowchart representation of a method 800 of presenting SRcontent in associated with recognized objects in accordance with someimplementations. In various implementations, the method 800 is performedby a device with non-transitory memory and one or more processorscoupled with the non-transitory memory (e.g., the controller 110 inFIGS. 1B and 2, the electronic device 120 in FIGS. 1A-1B and 3, or asuitable combination thereof). In some implementations, the method 800is performed by processing logic, including hardware, firmware,software, or a combination thereof. In some implementations, the method800 is performed by a processor executing code stored in anon-transitory computer-readable medium (e.g., a memory). Briefly, insome circumstances, the method 800 includes: obtaining image data froman image sensor; recognizing a portion of an object within the imagedata; obtaining SR content associated with the portion of the object;and displaying the SR content in association with the portion of theobject.

As represented by block 8-1, the method 800 includes obtaining imagedata from an image sensor. In some implementations, the electronicdevice 120 or a component thereof (e.g., the data obtainer 442) obtainsimage data from one or more external facing image sensors 314, whereinthe image data corresponds to images or a video stream capturing thephysical setting 105. In some implementations, the controller 110 or acomponent thereof (e.g., the data obtainer 242) obtains image data fromthe electronic device 120 that includes image data from one or moreexternal facing image sensors 314 of the electronic device 120, whereinthe image data corresponds to images or a video stream capturing thephysical setting 105. For example, the image data corresponds to imagesor a video feed from the external-facing cameras on the electronicdevice 120 (e.g., an HMD, tablet, mobile phone, or the like), cameraswithin the physical setting 105, or the like. With reference to FIG. 5A,for example, the image data corresponds to a live video feed of aportion of the physical setting 505 associated with the field-of-view520 of an external facing image sensor of the electronic device 120. Inthis example, the display 510 of the electronic device 120 shows theimage data including the portion of the physical setting 505 thatincludes the submarine model 512 sitting on the table 514.

As represented by block 8-2, the method 800 includes recognizing aportion of an object within the image data. In some implementations, thecontroller 110 or a component thereof (e.g., the object recognizer 243)recognizes one or more objects within the image data (e.g., the object160 within the physical setting 105 in FIGS. 1A-1B). In someimplementations, the electronic device 120 or a component thereof (e.g.,the optional object recognizer 341) recognizes one or more objectswithin the image data (e.g., the object 160 within the physical setting105 in FIGS. 1A-1B). For example, the controller 110 or the electronicdevice 120 performs object detection and recognition using object/shapeclassifiers on the image data in order to identify one or more objects.In some implementations, the controller 110 or the electronic device 120identifies the one or more objects based on depth data in place of or inconjunction to the image data. With reference to FIG. 5A, for example,the controller 110 or the electronic device 120 recognizes the submarinemodel 512 within the physical setting 505 and optionally also identifiesthe type of submarine associated with the submarine model 512.

In some implementations, the object corresponds to a scale-model orrepresentation of a real-life object such as a building, landscape,campus, vehicle, or the like. For example, in FIGS. 5A-5B, therecognized object corresponds to the model submarine 512. In someimplementations, the object corresponds to a real-life object such as abuilding, vehicle, ball, toy, furniture, or the like. For example, inFIG. 6A, the recognized object corresponds to the Eiffel Tower 612.

As represented by block 8-3, the method 800 includes obtaining (e.g.,receiving, retrieving, or generating) SR content associated with theportion of the object. In some implementations, the controller 110 or acomponent thereof (e.g., the SR content obtainer 246) obtains the SRcontent associated with the one or more objects recognized within theimage data.

In some implementations, as represented by block 8-3 a, the method 800includes receiving or retrieving the SR content from a library of SRcontent associated with the object. In some implementations, thecontroller 110 or a component thereof (e.g., the SR content obtainer246) obtains the SR content associated with the one or more objectsrecognized within the image data from a local library or a remotelibrary (e.g., a remote server, a third-party content provider, or thelike). In some implementations, each of the objects recognized withinimage data are associated with one or more instances of SR content(e.g., educational information, a cross-section thereof, associatedvideo content, and/or the like). With reference to FIGS. 5A-5B, forexample, the electronic device 120 superimposes the AR content 525 and575 on the model submarine 512 to show a cutaway view of the submarinetype.

In some implementations, each of the objects recognized within imagedata are associated with one or more instances of SR content forrespective portions of the objects (e.g., a top portion of an object isassociated with first SR content and a bottom portion of the object isassociated with second SR content). In some implementations, thecontroller 110 or a component thereof (e.g., the SR content manager 248)selects from various instances of SR content associated with arespective object recognized within the image data based on the portionof the object within the image data. For example, if a bow portion of aboat is detected within the image data, the electronic device presentsSR content associated with the bow. Continuing with this example,however, if an aft portion of a boat is detected within the image data,the electronic device presents SR content associated with the aft.

In some implementations, as represented by block 8-3 b, the method 800includes generating the SR content based on sensor data. In someimplementations, the controller 110 or a component thereof (e.g., the SRcontent obtainer 246) generates the SR content associated with the oneor more objects recognized within the image data based on environmentalsensor data or sensor data associated with the one or more objects. Insome implementations, the SR content is generated as a function ofsensors associated with the object (e.g., sensors within a car engine,which in turn lead to AR content providing a live simulation of the carengine). In some implementations, the SR content is generated as afunction of environmental sensors proximate to the object (e.g.,thermometers, wind gauges, rain gauges, humidity sensors, light sensors,electromagnetic wave sensors, and/or the like). According to someimplementations, a user is able to see AR content superimposed on anobject that is generated based on sensor data from the object and/orenvironmental sensors proximate to the object.

With reference to FIG. 6A, for example, the electronic device 120superimposes the AR content 625 a and 625 c on the Eiffel Tower 612corresponding to elevators moving within the Eiffel Tower based onsensor data from the Eiffel Tower 612. With reference to FIG. 6A, forexample, the electronic device 120 also displays AR content 625 bcorresponding to the crowd of people below the Eiffel Tower 612 orenvironmental statistics (e.g., wind speed, temperature, humidity, etc.)based on sensor data from local sensors or sensors within the physicalsetting 605. With reference to FIG. 6B, for example, the electronicdevice 120 superimposes the AR content 675 a and 675 b on the automobile662 corresponding to illustrations of the working transmission andengine of the automobile 662 based on sensor data from the automobile622.

In some implementations, as represented by block 8-3 c, the method 800includes generating the SR content based on video content. In someimplementations, the controller 110 or a component thereof (e.g., the SRcontent obtainer 246) generates the SR content based on video content.With reference to FIGS. 5A-5B, for example, the electronic device 120superimposes the AR content 525 and 575 on the model submarine 512 toshow a cutaway view of the model submarine 512, where the AR content 525and 575 is based on video content associated with the type of submarineassociated with the model submarine 512. With reference to FIG. 7, forexample, in state 710 (e.g., at time T), the user 150 is watching videocontent 705 (e.g., a TV episode or movie associated with a submarinecrew) on the display device 130. With continued reference to FIG. 7, instate 720 (e.g., at time T+1), the user 150 is wearing the electronicdevice 120 (e.g., an HMD) on his/her head and viewing an SRreconstruction 715 of the video content 705 being presented on and/oraround the model submarine 160.

As one example, a display device (e.g., TV or tablet) presents a movieassociated with a spaceship crew to a user while a physical model of thespaceship is located on the user's coffee table and the user is wearingan AR-enabled HMD with optical see-through. Continuing with thisexample, in response to a command from the user, playback of the movieon the display device stops and the AR-enabled HMD presents an SRreconstruction of the movie presented on and/or around the physicalmodel of the spaceship. As such, the user is able to seamlessly switchbetween viewing a movie on a display device and an SR reconstructionthereof superimposed on or around a related physical object or modelscene. In some implementations, the SR reconstruction of the videocontent is obtained (e.g., received or retrieved) from a library ofexisting SR content. For example, in some implementations, the SRreconstruction of the video content is generated based on the techniquesdescribed in U.S. Provisional Patent Application No. 62/620,334,Attorney Docket No. 196PR, filed Jan. 22, 2018, which is incorporatedherein in its entirety.

In some implementations, as represented by block 8-3 d, the SR contentis selected based on orientation of the device relative to the object.In some implementations, the controller 110 or a component thereof(e.g., the mapper and locator engine 244) determines the orientation ofthe electronic device 120 relative to one or more reference points(e.g., the object 160) in the physical setting (e.g., the center of massof the object 160 or another point on the object 160). In someimplementations, the electronic device 120 or a component thereof (e.g.,the orientation determiner 343) determines the orientation of theelectronic device 120 relative to one or more reference points (e.g., anobject) in the physical setting (e.g., the center of mass of or anotherpoint on the object 160). For example, in some implementations, theorientation of the electronic device 120 is determined relative to theobject based on the techniques described in U.S. Provisional PatentApplication No. 62/556,849, Attorney Docket No. 173PR, filed Sep. 11,2017, which is incorporated herein in its entirety. In someimplementations, the controller 110 or a component thereof (e.g., the SRcontent manager 248) selects from various instances of SR contentassociated with a respective object recognized within the image databased on the orientation of the electronic device 120 relative to therespective object.

For example, as shown in FIGS. 5A-5B, the AR content changes based onthe orientation of the electronic device 120 relative to the object. InFIG. 5A, the electronic device 120 superimposes the AR content 525 onthe model submarine 512 based on the side or perspective orientation ofthe electronic device 120 relative to the model submarine 512. Incontrast, in FIG. 5B, the electronic device 120 superimposes the ARcontent 575 on the model submarine 512 based on the top-down orientationof the electronic device 120 relative to the model submarine 512.

As represented by block 8-4, the method 800 includes displaying the SRcontent in association with the portion of the object. In someimplementations, the controller 110 or a component thereof (e.g., the SRcontent manager 248) coordinates the presentation of the SR content bythe electronic device 120. In some implementations, the controller 110or a component thereof (e.g., the data transmitter 250) providespresentation data associated with the SR content to the electronicdevice 120 for presentation thereby. In some implementations, theelectronic device 120 or a component thereof (e.g., the SR presenter344) presents the SR content in association with the object via the oneor more displays 312.

In some implementations, the SR content is overlaid or superimposed onthe object. In some implementations, the SR content is displayed aroundor adjacent to the object. In some implementations, once the object isrecognized, the electronic device 120 displays a subtle affordance orprompt that is, in turn, selected to initiate presentation of the SRcontent. In some implementations, the user is also able to select frommultiple instances of SR content associated with the recognized object.In some implementations, the SR content is interactive, in which theuser is able to interact with the with the SR content. In turn, the SRcontent is updated based on the user interactions (e.g., zoom in/out,spin, flip, move, disassemble, reassemble, etc. actions). In someimplementations, as the user interacts with the SR content, theelectronic device 120 provides audio, haptic, skin shear, temperature,or the like feedback.

In some implementations, as represented by block 8-4 a, the SR contentis superimposed on the object. For example, the SR content is overlaidor superimposed on the object as such the SR content may show a workingengine of the object car, multiple decks of the object spaceship, orproposed modifications/additions to the object. In another example, theSR content is displayed transparently to provide an “x-ray” view intothe object. With reference to FIGS. 5A-5B, for example, the electronicdevice 120 superimposes the AR content 525 and 575 on the modelsubmarine 512 to show a cutaway view of the submarine type. In anotherexample, the SR content shows an “x-ray” view of a human body, acontainer of food, a consumer product, a household appliance, acollectible miniature, or the like (e.g., the recognized object). In yetanother example, the SR content corresponds to an original or restoredstate of an ancient object (e.g., antique furniture, historic relic,ancient ruins, or the like).

In some implementations, as represented by block 8-4 b, the SR contentis displayed around or adjacent to the object. For example, the SRcontent is displayed around or adjacent to the object such aselectromagnetic waves penetrating a building or wind gusts striking atree or building. With reference to FIG. 6A, for example, the electronicdevice 120 presented the AR content 625 b that corresponds to the crowdof people below the Eiffel Tower 612 or environmental statistics (e.g.,wind speed, temperature, humidity, etc.) based on sensor data fromsensors associated with the electronic device 120, sensors associatedwith the Eiffel Tower 612, or sensors within the physical setting 605.

While various aspects of implementations within the scope of theappended claims are described above, it should be apparent that thevarious features of implementations described above may be embodied in awide variety of forms and that any specific structure and/or functiondescribed above is merely illustrative. Based on the present disclosureone skilled in the art should appreciate that an aspect described hereinmay be implemented independently of any other aspects and that two ormore of these aspects may be combined in various ways. For example, anapparatus may be implemented and/or a method may be practiced using anynumber of the aspects set forth herein. In addition, such an apparatusmay be implemented and/or such a method may be practiced using otherstructure and/or functionality in addition to or other than one or moreof the aspects set forth herein.

It will also be understood that, although the terms “first,” “second,”etc. may be used herein to describe various elements, these elementsshould not be limited by these terms. These terms are only used todistinguish one element from another. For example, a first node could betermed a second node, and, similarly, a second node could be termed afirst node, which changing the meaning of the description, so long asall occurrences of the “first node” are renamed consistently and alloccurrences of the “second node” are renamed consistently. The firstnode and the second node are both nodes, but they are not the same node.

The terminology used herein is for the purpose of describing particularembodiments only and is not intended to be limiting of the claims. Asused in the description of the embodiments and the appended claims, thesingular forms “a,” “an,” and “the” are intended to include the pluralforms as well, unless the context clearly indicates otherwise. It willalso be understood that the term “and/or” as used herein refers to andencompasses any and all possible combinations of one or more of theassociated listed items. It will be further understood that the terms“comprises” and/or “comprising,” when used in this specification,specify the presence of stated features, integers, steps, operations,elements, and/or components, but do not preclude the presence oraddition of one or more other features, integers, steps, operations,elements, components, and/or groups thereof.

As used herein, the term “if” may be construed to mean “when” or “upon”or “in response to determining” or “in accordance with a determination”or “in response to detecting,” that a stated condition precedent istrue, depending on the context. Similarly, the phrase “if it isdetermined [that a stated condition precedent is true]” or “if [a statedcondition precedent is true]” or “when [a stated condition precedent istrue]” may be construed to mean “upon determining” or “in response todetermining” or “in accordance with a determination” or “upon detecting”or “in response to detecting” that the stated condition precedent istrue, depending on the context.

What is claimed is:
 1. A method comprising: at a computing systemincluding non-transitory memory and one or more processors, wherein thecomputing system is communicatively coupled to a display device, one ormore input devices, and an image sensor: obtaining, via the imagesensor, image data associated with a physical setting; recognizing afirst physical object and a second physical object within the image dataassociated with the physical setting; determining a first camera poseassociated with the image data relative to the first and second physicalobjects; obtaining a first synthesized reality (SR) object associatedwith the first physical object based on the first camera pose relativeto the first physical object; obtaining a second SR object associatedwith the second physical object based on the first camera pose relativeto the second physical object, wherein the second SR object is differentfrom the first SR object causing presentation of the first SR object inassociation with the first physical object and the second SR object inassociation with the second physical object via the display device. 2.The method of claim 1, wherein obtaining the first SR object associatedwith the first physical object based on the first camera pose relativeto the first physical object includes: obtaining a third SR objectassociated with the first physical object in accordance with adetermination that the first camera pose relative to the first physicalobject correspond to a first point-of-view (POV) of the first physicalobject; and obtaining a fourth SR object associated with the firstphysical object, wherein the fourth SR object is different from thethird SR object, in accordance with a determination that the firstcamera pose relative to the first physical object correspond to a secondPOV of the first physical object different from the first POV of thefirst physical object.
 3. The method of claim 1, further comprising:detecting a change from the first camera pose to a second camera poseassociated with the image data relative to the first and second physicalobjects; and in response to detecting the change to the second camerapose: obtaining a third SR object associated with the first physicalobject based on the second camera pose relative to the first physicalobject; obtaining a fourth SR object associated with the second physicalobject based on the second camera pose relative to the second physicalobject, wherein the fourth SR object is different from the first thirdobject causing presentation of the third SR object in association withthe first physical object and the fourth SR object in association withthe second physical object via the display device.
 4. The method ofclaim 1, wherein the first SR object is overlaid or superimposed on aportion of the first physical object, and wherein the second SR objectis overlaid or superimposed on a portion of the second physical object.5. The method of claim 1, wherein the first SR object is displayedaround or adjacent to a portion of the first physical object, andwherein the second SR object is displayed around or adjacent to aportion of the second physical object.
 6. The method of claim 1, whereinthe first and second SR objects are selected from a library ofpre-existing SR content associated with the first and second physicalobjects.
 7. The method of claim 1, wherein the first and second SRobjects are generated based on sensor data associated with the first andsecond physical objects.
 8. The method of claim 1, wherein the first andsecond SR objects are generated based on video content associated withthe first and second physical objects.
 9. The method of claim 1, furthercomprising: displaying a first affordance provided to initiate displayof the first SR object in response to obtaining the first SR objectassociated with the first physical object based on the first camera poserelative to the first physical object, wherein the first SR object isdisplayed in association with the first physical object in response todetecting selection of the first affordance; and displaying a secondaffordance provided to initiate display of the second SR object inresponse to obtaining the second SR object associated with the secondphysical object based on the first camera pose relative to the secondphysical object, wherein the second SR object is displayed inassociation with the second physical object in response to detectingselection of the second affordance.
 10. A computing system comprising:one or more processors; a communication interface for communicating witha display device, one or more input devices, and an image sensor; anon-transitory memory; and one or more programs stored in thenon-transitory memory, which, when executed by the one or moreprocessors, cause the computing system to: obtain, via the image sensor,image data associated with a physical setting; recognize a firstphysical object and a second physical object within the image dataassociated with the physical setting; determine a first camera poseassociated with the image data relative to the first and second physicalobjects; obtain a first synthesized reality (SR) object associated withthe first physical object based on the first camera pose relative to thefirst physical object; obtain a second SR object associated with thesecond physical object based on the first camera pose relative to thesecond physical object, wherein the second SR object is different fromthe first SR object cause presentation of the first SR object inassociation with the first physical object and the second SR object inassociation with the second physical object via the display device. 11.The computing system of claim 10, wherein obtaining the first SR objectassociated with the first physical object based on the first camera poserelative to the first physical object includes: obtaining a third SRobject associated with the first physical object in accordance with adetermination that the first camera pose relative to the first physicalobject correspond to a first point-of-view (POV) of the first physicalobject; and obtaining a fourth SR object associated with the firstphysical object, wherein the fourth SR object is different from thethird SR object, in accordance with a determination that the firstcamera pose relative to the first physical object correspond to a secondPOV of the first physical object different from the first POV of thefirst physical object.
 12. The computing system of claim 10, wherein theone or more programs further cause the computing system to: detect achange from the first camera pose to a second camera pose associatedwith the image data relative to the first and second physical objects;and in response to detecting the change to the second camera pose:obtain a third SR object associated with the first physical object basedon the second camera pose relative to the first physical object; obtaina fourth SR object associated with the second physical object based onthe second camera pose relative to the second physical object, whereinthe fourth SR object is different from the first third object causepresentation of the third SR object in association with the firstphysical object and the fourth SR object in association with the secondphysical object via the display device.
 13. The computing system ofclaim 10, wherein the first SR object is overlaid or superimposed on aportion of the first physical object, and wherein the second SR objectis overlaid or superimposed on a portion of the second physical object.14. The computing system of claim 10, wherein the first SR object isdisplayed around or adjacent to a portion of the first physical object,and wherein the second SR object is displayed around or adjacent to aportion of the second physical object.
 15. The computing system of claim10, wherein the one or more programs further cause the computing systemto: display, via the display device, a first affordance provided toinitiate display of the first SR object in response to obtaining thefirst SR object associated with the first physical object based on thefirst camera pose relative to the first physical object, wherein thefirst SR object is displayed in association with the first physicalobject in response to detecting selection of the first affordance; anddisplay, via the display device, a second affordance provided toinitiate display of the second SR object in response to obtaining thesecond SR object associated with the second physical object based on thefirst camera pose relative to the second physical object, wherein thesecond SR object is displayed in association with the second physicalobject in response to detecting selection of the second affordance. 16.A non-transitory memory storing one or more programs, which, whenexecuted by one or more processors of a computing system with acommunication interface for communicating with a display device, one ormore input devices, and an image sensor, cause the computing system to:obtain, via the image sensor, image data associated with a physicalsetting; recognize a first physical object and a second physical objectwithin the image data associated with the physical setting; determine afirst camera pose associated with the image data relative to the firstand second physical objects; obtain a first synthesized reality (SR)object associated with the first physical object based on the firstcamera pose relative to the first physical object; obtain a second SRobject associated with the second physical object based on the firstcamera pose relative to the second physical object, wherein the secondSR object is different from the first SR object cause presentation ofthe first SR object in association with the first physical object andthe second SR object in association with the second physical object viathe display device.
 17. The non-transitory memory of claim 16, whereinobtaining the first SR object associated with the first physical objectbased on the first camera pose relative to the first physical objectincludes: obtaining a third SR object associated with the first physicalobject in accordance with a determination that the first camera poserelative to the first physical object correspond to a firstpoint-of-view (POV) of the first physical object; and obtaining a fourthSR object associated with the first physical object, wherein the fourthSR object is different from the third SR object, in accordance with adetermination that the first camera pose relative to the first physicalobject correspond to a second POV of the first physical object differentfrom the first POV of the first physical object.
 18. The non-transitorymemory of claim 16, wherein the one or more programs further cause thecomputing system to: detect a change from the first camera pose to asecond camera pose associated with the image data relative to the firstand second physical objects; and in response to detecting the change tothe second camera pose: obtain a third SR object associated with thefirst physical object based on the second camera pose relative to thefirst physical object; obtain a fourth SR object associated with thesecond physical object based on the second camera pose relative to thesecond physical object, wherein the fourth SR object is different fromthe first third object cause presentation of the third SR object inassociation with the first physical object and the fourth SR object inassociation with the second physical object via the display device. 19.The non-transitory memory of claim 16, wherein the first SR object isoverlaid or superimposed on a portion of the first physical object, andwherein the second SR object is overlaid or superimposed on a portion ofthe second physical object.
 20. The non-transitory memory of claim 16,wherein the first SR object is displayed around or adjacent to a portionof the first physical object, and wherein the second SR object isdisplayed around or adjacent to a portion of the second physical object.21. The non-transitory memory of claim 16, wherein the one or moreprograms further cause the computing system to: display, via the displaydevice, a first affordance provided to initiate display of the first SRobject in response to obtaining the first SR object associated with thefirst physical object based on the first camera pose relative to thefirst physical object, wherein the first SR object is displayed inassociation with the first physical object in response to detectingselection of the first affordance; and display, via the display device,a second affordance provided to initiate display of the second SR objectin response to obtaining the second SR object associated with the secondphysical object based on the first camera pose relative to the secondphysical object, wherein the second SR object is displayed inassociation with the second physical object in response to detectingselection of the second affordance.